[Helm] Fix: inject flag-based env into ConfigMap when configuration.enabled=true by win5923 · Pull Request #4270 · ray-project/kuberay

win5923 · 2025-12-14T13:38:18Z

Why are these changes needed?

Some helm values, e.g. leaderElectionEnabled and EnableMetrics, are passed as flags, see here:

kuberay/helm-chart/kuberay-operator/templates/deployment.yaml

Lines 126 to 130 in 9e8a2c0

    
           {{- if hasKey .Values "leaderElectionEnabled" -}} 
        
           {{- $argList = append $argList (printf "--enable-leader-election=%t" .Values.leaderElectionEnabled) -}} 
        
           {{- end -}} 
        
           {{- if and (hasKey .Values "metrics") (hasKey .Values.metrics "enabled") }} 
        
           {{- $argList = append $argList (printf "--enable-metrics=%t" .Values.metrics.enabled) -}}

Then these flags get ignored when configuration.enabled=true, see here:

kuberay/ray-operator/main.go

Lines 117 to 141 in 9e8a2c0

    
           if configFile != "" { 
        
           	var err error 
        
           	configData, err := os.ReadFile(configFile) 
        
           	exitOnError(err, "failed to read config file") 
        
           	config, err = decodeConfig(configData, scheme) 
        
           	exitOnError(err, "failed to decode config file") 
        
           } else { 
        
           	config.MetricsAddr = metricsAddr 
        
           	config.ProbeAddr = probeAddr 
        
           	config.EnableLeaderElection = &enableLeaderElection 
        
           	config.LeaderElectionNamespace = leaderElectionNamespace 
        
           	config.ReconcileConcurrency = reconcileConcurrency 
        
           	config.WatchNamespace = watchNamespace 
        
           	config.LogFile = logFile 
        
           	config.LogFileEncoder = logFileEncoder 
        
           	config.LogStdoutEncoder = logStdoutEncoder 
        
           	config.EnableBatchScheduler = enableBatchScheduler 
        
           	config.BatchScheduler = batchScheduler 
        
           	config.UseKubernetesProxy = useKubernetesProxy 
        
           	config.DeleteRayJobAfterJobFinishes = os.Getenv(utils.DELETE_RAYJOB_CR_AFTER_JOB_FINISHES) == "true" 
        
           	config.EnableMetrics = enableMetrics 
        
           	config.QPS = &qps 
        
           	config.Burst = &burst 
        
           }

kuberay/helm-chart/kuberay-operator/values.yaml

Lines 104 to 107 in 9e8a2c0

    
           # Configuration for the KubeRay operator. 
        
           configuration: 
        
             # -- Whether to enable the configuration feature. If enabled, a ConfigMap will be created and mounted to the operator. 
        
             enabled: false

kuberay/helm-chart/kuberay-operator/templates/deployment.yaml

Lines 86 to 91 in 9e8a2c0

    
           args: 
        
             {{- $argList := list -}} 
        
             {{- if .Values.configuration.enabled -}} 
        
             {{- $argList = append $argList "--config" -}} 
        
             {{- $argList = append $argList "/etc/kuberay/config.yaml" -}} 
        
             {{- end -}}

kuberay/helm-chart/kuberay-operator/templates/configmap.yaml

Lines 1 to 20 in 9e8a2c0

    
           {{- if .Values.configuration.enabled }} 
        
           apiVersion: v1 
        
           kind: ConfigMap 
        
           metadata: 
        
             name: {{ include "kuberay-operator.deployment.name" . }}-config 
        
             namespace: {{ .Release.Namespace }} 
        
             labels: 
        
               {{- include "kuberay-operator.labels" . | nindent 4 }} 
        
               {{- with .Values.labels }} 
        
               {{- toYaml . | nindent 4 }} 
        
               {{- end }} 
        
           data: 
        
             config.yaml: | 
        
               apiVersion: config.ray.io/v1alpha1 
        
               kind: Configuration 
        
               {{- if .Values.configuration.defaultContainerEnvs }} 
        
               defaultContainerEnvs: 
        
               {{- toYaml .Values.configuration.defaultContainerEnvs | nindent 4 }} 
        
               {{- end }} 
        
           {{- end }}

This PR includes all flag-based configuration options in the ConfigMap template, ensuring they are properly read from helm values and injected into the ConfigMap.

When configuration.enabled = true, the KubeRay Operator would read all flag-based envs from the ConfigMap, and these values would take precedence over the flag-based configuration.
When configuration.enabled = false and the relevant flag-based options are explicitly set in values.yaml (e.g. metrics.enabled = true), the operator would preserving the current flag-based behavior.

Result:

Related issue number

Closes #4268

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

…s enabled Signed-off-by: win5923 <ken89@kimo.com>

helm-chart/kuberay-operator/templates/configmap.yaml

seanlaii

Thanks! Overall this looks great to me.
Aside from the comment below, I have one minor suggestion: should we consider avoiding setting this flag?

helm-chart/kuberay-operator/templates/configmap.yaml

win5923 · 2025-12-19T12:05:23Z

I have one minor suggestion: should we consider avoiding setting this flag?

I think both are fine, This flag is primarily designed for complex structures that cannot be easily passed via command-line flags (like defaultContainerEnvs, headSidecarContainers).

Signed-off-by: win5923 <ken89@kimo.com>

seanlaii · 2025-12-19T15:17:57Z

I think both are fine, This flag is primarily designed for complex structures that cannot be easily passed via command-line flags (like defaultContainerEnvs, headSidecarContainers).

Sorry for the confusion. What I mean is should we consider avoiding setting the command line flags when they enable the config map mode?
IIUC, the command line flags will still be set when enabling the config map mode.
Maybe the users will be confused when they still see the command line flags while using the config map mode.

win5923 · 2025-12-19T18:09:06Z

Sorry for the confusion. What I mean is should we consider avoiding setting the command line flags when they enable the config map mode?
IIUC, the command line flags will still be set when enabling the config map mode.
Maybe the users will be confused when they still see the command line flags while using the config map mode.

Oh, Thanks for catching this. I think we should consider avoiding setting the command line flags when they enable the config map mode.

Signed-off-by: win5923 <ken89@kimo.com>

seanlaii · 2025-12-21T17:46:27Z

helm-chart/kuberay-operator/templates/deployment.yaml

            {{- end -}}
            {{- end -}}
+            {{- end -}}
+            {{- $argList = append $argList (include "kuberay.featureGates" . | trim) -}}


Thanks! To confirm, featureGates is not supported in config map, is it correct?

seanlaii

Thanks! LGTM!

Future-Outlier

cc @troychiu @fsNick @machichima to help review.
plz provide screenshot if possible, thank you!

helm-chart/kuberay-operator/templates/_helpers.tpl

helm-chart/kuberay-operator/values.yaml

Signed-off-by: win5923 <ken89@kimo.com>

machichima

LGTM!

Future-Outlier

high level looks good to me, trust @seanlaii and @machichima 's review

win5923 marked this pull request as ready for review December 14, 2025 13:38

win5923 marked this pull request as draft December 14, 2025 13:51

win5923 marked this pull request as ready for review December 14, 2025 14:38

[Helm] Fix: inject flag-based env into ConfigMap when configuration i…

418d799

…s enabled Signed-off-by: win5923 <ken89@kimo.com>

win5923 force-pushed the helm-configmap branch from 8c2c7e6 to 418d799 Compare December 14, 2025 17:23

win5923 commented Dec 14, 2025

View reviewed changes

helm-chart/kuberay-operator/templates/configmap.yaml Outdated Show resolved Hide resolved

seanlaii reviewed Dec 18, 2025

View reviewed changes

helm-chart/kuberay-operator/templates/configmap.yaml Outdated Show resolved Hide resolved

Use named template

4d4f437

Signed-off-by: win5923 <ken89@kimo.com>

win5923 added 2 commits December 20, 2025 02:12

Exclude duplicate command-line flags when ConfigMap mode is enabled

fe52eb4

Signed-off-by: win5923 <ken89@kimo.com>

Merge branch 'upstream-master' into helm-configmap

7bea5fe

seanlaii reviewed Dec 21, 2025

View reviewed changes

seanlaii approved these changes Dec 21, 2025

View reviewed changes

Future-Outlier added this to @Future-Outlier's kuberay project Jan 5, 2026

Future-Outlier moved this to In Progress in @Future-Outlier's kuberay project Jan 5, 2026

Future-Outlier self-assigned this Jan 12, 2026

Future-Outlier reviewed Jan 15, 2026

View reviewed changes

machichima reviewed Jan 18, 2026

View reviewed changes

helm-chart/kuberay-operator/templates/_helpers.tpl Outdated Show resolved Hide resolved

helm-chart/kuberay-operator/values.yaml Outdated Show resolved Hide resolved

Reslove comments

da5d9c0

Signed-off-by: win5923 <ken89@kimo.com>

machichima approved these changes Jan 19, 2026

View reviewed changes

Merge branch 'upstream-master' into helm-configmap

86349f3

Future-Outlier approved these changes Jan 20, 2026

View reviewed changes

Future-Outlier moved this from In Progress to can be merged in @Future-Outlier's kuberay project Jan 20, 2026

rueian approved these changes Jan 20, 2026

View reviewed changes

rueian merged commit df6eff5 into ray-project:master Jan 20, 2026
29 checks passed

github-project-automation bot moved this from can be merged to Done in @Future-Outlier's kuberay project Jan 20, 2026

win5923 deleted the helm-configmap branch January 20, 2026 18:19

	{{- if hasKey .Values "leaderElectionEnabled" -}}
	{{- $argList = append $argList (printf "--enable-leader-election=%t" .Values.leaderElectionEnabled) -}}
	{{- end -}}
	{{- if and (hasKey .Values "metrics") (hasKey .Values.metrics "enabled") }}
	{{- $argList = append $argList (printf "--enable-metrics=%t" .Values.metrics.enabled) -}}

	if configFile != "" {
	var err error
	configData, err := os.ReadFile(configFile)
	exitOnError(err, "failed to read config file")

	config, err = decodeConfig(configData, scheme)
	exitOnError(err, "failed to decode config file")
	} else {
	config.MetricsAddr = metricsAddr
	config.ProbeAddr = probeAddr
	config.EnableLeaderElection = &enableLeaderElection
	config.LeaderElectionNamespace = leaderElectionNamespace
	config.ReconcileConcurrency = reconcileConcurrency
	config.WatchNamespace = watchNamespace
	config.LogFile = logFile
	config.LogFileEncoder = logFileEncoder
	config.LogStdoutEncoder = logStdoutEncoder
	config.EnableBatchScheduler = enableBatchScheduler
	config.BatchScheduler = batchScheduler
	config.UseKubernetesProxy = useKubernetesProxy
	config.DeleteRayJobAfterJobFinishes = os.Getenv(utils.DELETE_RAYJOB_CR_AFTER_JOB_FINISHES) == "true"
	config.EnableMetrics = enableMetrics
	config.QPS = &qps
	config.Burst = &burst
	}

	# Configuration for the KubeRay operator.
	configuration:
	# -- Whether to enable the configuration feature. If enabled, a ConfigMap will be created and mounted to the operator.
	enabled: false

	args:
	{{- $argList := list -}}
	{{- if .Values.configuration.enabled -}}
	{{- $argList = append $argList "--config" -}}
	{{- $argList = append $argList "/etc/kuberay/config.yaml" -}}
	{{- end -}}

	{{- if .Values.configuration.enabled }}
	apiVersion: v1
	kind: ConfigMap
	metadata:
	name: {{ include "kuberay-operator.deployment.name" . }}-config
	namespace: {{ .Release.Namespace }}
	labels:
	{{- include "kuberay-operator.labels" . \| nindent 4 }}
	{{- with .Values.labels }}
	{{- toYaml . \| nindent 4 }}
	{{- end }}
	data:
	config.yaml: \|
	apiVersion: config.ray.io/v1alpha1
	kind: Configuration
	{{- if .Values.configuration.defaultContainerEnvs }}
	defaultContainerEnvs:
	{{- toYaml .Values.configuration.defaultContainerEnvs \| nindent 4 }}
	{{- end }}
	{{- end }}

Conversation

win5923 commented Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

Uh oh!

seanlaii left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

win5923 commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seanlaii commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

win5923 commented Dec 19, 2025

Uh oh!

seanlaii Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

win5923 Dec 21, 2025

Choose a reason for hiding this comment

Uh oh!

seanlaii left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

machichima left a comment

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Comments

win5923 commented Dec 14, 2025 •

edited

Loading

win5923 commented Dec 19, 2025 •

edited

Loading

seanlaii commented Dec 19, 2025 •

edited

Loading

seanlaii left a comment •

edited

Loading

Future-Outlier left a comment •

edited

Loading